36 research outputs found
Proximal Multitask Learning over Networks with Sparsity-inducing Coregularization
In this work, we consider multitask learning problems where clusters of nodes
are interested in estimating their own parameter vector. Cooperation among
clusters is beneficial when the optimal models of adjacent clusters have a good
number of similar entries. We propose a fully distributed algorithm for solving
this problem. The approach relies on minimizing a global mean-square error
criterion regularized by non-differentiable terms to promote cooperation among
neighboring clusters. A general diffusion forward-backward splitting strategy
is introduced. Then, it is specialized to the case of sparsity promoting
regularizers. A closed-form expression for the proximal operator of a weighted
sum of -norms is derived to achieve higher efficiency. We also provide
conditions on the step-sizes that ensure convergence of the algorithm in the
mean and mean-square error sense. Simulations are conducted to illustrate the
effectiveness of the strategy
Quantization for decentralized learning under subspace constraints
In this paper, we consider decentralized optimization problems where agents
have individual cost functions to minimize subject to subspace constraints that
require the minimizers across the network to lie in low-dimensional subspaces.
This constrained formulation includes consensus or single-task optimization as
special cases, and allows for more general task relatedness models such as
multitask smoothness and coupled optimization. In order to cope with
communication constraints, we propose and study an adaptive decentralized
strategy where the agents employ differential randomized quantizers to compress
their estimates before communicating with their neighbors. The analysis shows
that, under some general conditions on the quantization noise, and for
sufficiently small step-sizes , the strategy is stable both in terms of
mean-square error and average bit rate: by reducing , it is possible to
keep the estimation errors small (on the order of ) without increasing
indefinitely the bit rate as . Simulations illustrate the
theoretical findings and the effectiveness of the proposed approach, revealing
that decentralized learning is achievable at the expense of only a few bits
Distributed adaptive estimation over multitask networks
Lâapprentissage adaptatif distribuĂ© sur les rĂ©seaux permet Ă un ensemble dâagents de rĂ©soudre des problĂšmes dâestimation de paramĂštres en ligne en se basant sur des calculs locaux et sur des Ă©changes locaux avec les voisins immĂ©diats. La littĂ©rature sur lâestimation distribuĂ©e considĂšre essentiellement les problĂšmes Ă simple tĂąche, oĂč les agents disposant de fonctions objectives sĂ©parables doivent converger vers un vecteur de paramĂštres commun. Cependant, dans de nombreuses applications nĂ©cessitant des modĂšles plus complexes et des algorithmes plus flexibles, les agents ont besoin dâestimer et de suivre plusieurs vecteurs de paramĂštres simultanĂ©ment. Nous appelons ce type de rĂ©seau, oĂč les agents doivent estimer plusieurs vecteurs de paramĂštres, rĂ©seau multitĂąche. Bien que les agents puissent avoir diffĂ©rentes tĂąches Ă rĂ©soudre, ils peuvent capitaliser sur le transfert inductif entre eux afin dâamĂ©liorer les performances de leurs estimĂ©s. Le but de cette thĂšse est de proposer et dâĂ©tudier de nouveaux algorithmes dâestimation distribuĂ©e sur les rĂ©seaux multitĂąches. Dans un premier temps, nous prĂ©sentons lâalgorithme diffusion LMS qui est une stratĂ©gie efficace pour rĂ©soudre les problĂšmes dâestimation Ă simple-tĂąche et nous Ă©tudions thĂ©oriquement ses performances lorsquâil est mis en oeuvre dans un environnement multitĂąche et que les communications entre les noeuds sont bruitĂ©es. Ensuite, nous prĂ©sentons une stratĂ©gie de clustering non-supervisĂ© permettant de regrouper les noeuds rĂ©alisant une mĂȘme tĂąche en clusters, et de restreindre les Ă©changes dâinformation aux seuls noeuds dâun mĂȘme clusterDistributed adaptive learning allows a collection of interconnected agents to perform parameterestimation tasks from streaming data by relying solely on local computations and interactions with immediate neighbors. Most prior literature on distributed inference is concerned with single-task problems, where agents with separable objective functions need to agree on a common parameter vector. However, many network applications require more complex models and flexible algorithms than single-task implementations since their agents involve the need to estimate and track multiple objectives simultaneously. Networks of this kind, where agents need to infer multiple parameter vectors, are referred to as multitask networks. Although agents may generally have distinct though related tasks to perform, they may still be able to capitalize on inductive transfer between them to improve their estimation accuracy. This thesis is intended to bring forth advances on distributed inference over multitask networks. First, we present the well-known diffusion LMS strategies to solve single-task estimation problems and we assess their performance when they are run in multitask environments in the presence of noisy communication links. An improved strategy allowing the agents to adapt their cooperation to neighbors sharing the same objective is presented in order to attain improved learningand estimation over networks. Next, we consider the multitask diffusion LMS strategy which has been proposed to solve multitask estimation problems where the network is decomposed into clusters of agents seeking differen
Estimation distribuée adaptative sur les réseaux multitùches
Distributed adaptive learning allows a collection of interconnected agents to perform parameterestimation tasks from streaming data by relying solely on local computations and interactions with immediate neighbors. Most prior literature on distributed inference is concerned with single-task problems, where agents with separable objective functions need to agree on a common parameter vector. However, many network applications require more complex models and flexible algorithms than single-task implementations since their agents involve the need to estimate and track multiple objectives simultaneously. Networks of this kind, where agents need to infer multiple parameter vectors, are referred to as multitask networks. Although agents may generally have distinct though related tasks to perform, they may still be able to capitalize on inductive transfer between them to improve their estimation accuracy. This thesis is intended to bring forth advances on distributed inference over multitask networks. First, we present the well-known diffusion LMS strategies to solve single-task estimation problems and we assess their performance when they are run in multitask environments in the presence of noisy communication links. An improved strategy allowing the agents to adapt their cooperation to neighbors sharing the same objective is presented in order to attain improved learningand estimation over networks. Next, we consider the multitask diffusion LMS strategy which has been proposed to solve multitask estimation problems where the network is decomposed into clusters of agents seeking differentLâapprentissage adaptatif distribuĂ© sur les rĂ©seaux permet Ă un ensemble dâagents de rĂ©soudre des problĂšmes dâestimation de paramĂštres en ligne en se basant sur des calculs locaux et sur des Ă©changes locaux avec les voisins immĂ©diats. La littĂ©rature sur lâestimation distribuĂ©e considĂšre essentiellement les problĂšmes Ă simple tĂąche, oĂč les agents disposant de fonctions objectives sĂ©parables doivent converger vers un vecteur de paramĂštres commun. Cependant, dans de nombreuses applications nĂ©cessitant des modĂšles plus complexes et des algorithmes plus flexibles, les agents ont besoin dâestimer et de suivre plusieurs vecteurs de paramĂštres simultanĂ©ment. Nous appelons ce type de rĂ©seau, oĂč les agents doivent estimer plusieurs vecteurs de paramĂštres, rĂ©seau multitĂąche. Bien que les agents puissent avoir diffĂ©rentes tĂąches Ă rĂ©soudre, ils peuvent capitaliser sur le transfert inductif entre eux afin dâamĂ©liorer les performances de leurs estimĂ©s. Le but de cette thĂšse est de proposer et dâĂ©tudier de nouveaux algorithmes dâestimation distribuĂ©e sur les rĂ©seaux multitĂąches. Dans un premier temps, nous prĂ©sentons lâalgorithme diffusion LMS qui est une stratĂ©gie efficace pour rĂ©soudre les problĂšmes dâestimation Ă simple-tĂąche et nous Ă©tudions thĂ©oriquement ses performances lorsquâil est mis en oeuvre dans un environnement multitĂąche et que les communications entre les noeuds sont bruitĂ©es. Ensuite, nous prĂ©sentons une stratĂ©gie de clustering non-supervisĂ© permettant de regrouper les noeuds rĂ©alisant une mĂȘme tĂąche en clusters, et de restreindre les Ă©changes dâinformation aux seuls noeuds dâun mĂȘme cluste
Jungian Metaphor within the Selected Works of H.D., W.B. Yeats, and James Joyce
This thesis will argue for the centrality of Carl Jungâs theory of individuation and alchemy in modernist poetics. Jungâs position in this context is relatively unexamined, and published works often represent misreadings and distortions of Jungâs theory in this field; in particular, Jungian literary criticismâs use of Jungâs theories of the anima, the collective unconscious, alchemy, and individuation. The specific works discussed in this novel context are H.D.âs Trilogy, Yeatsâs poems and A Vision, and Joyceâs A Portrait of the Artist as a Young Man, Ulysses, and Finnegans Wake. These works will be read in light of Jungâs central theme of alchemy, which is a metaphor for âindividuationâ, or personal development, a process attained through an âalchemical marriageâ, or union of antinomial (âmaleâ and âfemaleâ) elements of the psyche. In the works of H.D., Yeats, and Joyce, there are attempts at developing a related alchemical model, a Jungian poetics, which serves to expand a readerâs understanding of modernist uses of language.
While critical reading of Jung and his revisionists establishes the ground for this thesisâs discussion of the alchemical theme of transformation, the first chapter considers the personal philosophies of the writers pertinent to this study, surveys modernist poetics, and pays attention to Arthur Rimbaudâs âalchemy of the wordâ. The following chapters observe aspects of a Jungian poetics in each of H.D., Yeats, and Joyceâs works, examining H.D.âs verbal alchemy, Yeatsâs visionary alchemy, and Joyceâs textual individuation. First, H.D. is shown to adopt alchemy as a style, through which she aims to recreate a feminine principle and establish a new mythos. The following chapter critically considers a Jungian reading of Yeatsâs works in terms of the âevocativeâ nature of poetry, as a manifestation of creativity, capable of giving the individual access to a collective unconscious. Finally, the fourth chapter continues the examination of the central alchemical theme and writing style in Joyceâs novels, through which he aims to transform both text and protagonist
Adaptation and Learning Over Networks Under Subspace Constraints & x2014;Part II: Performance Analysis
Part & x00A0;I of this paper considered optimization problems over networks where agents have individual objectives to meet, or individual parameter vectors to estimate, subject to subspace constraints that require the objectives across the network to lie in low-dimensional subspaces. Starting from the centralized projected gradient descent, an iterative and distributed solution was proposed that responds to streaming data and employs stochastic approximations in place of actual gradient vectors, which are generally unavailable. We examined the second-order stability of the learning algorithm and we showed that, for small step-sizes , the proposed strategy leads to small estimation errors on the order of . This Part & x00A0;II examines steady-state performance. The results reveal explicitly the influence of the gradient noise, data characteristics, and subspace constraints, on the network performance. The results also show that in the small step-size regime, the iterates generated by the distributed algorithm achieve the centralized steady-state performance
Distributed Inference Over Networks Under Subspace Constraints
This paper considers optimization problems over networks where agents have individual objectives to meet, or individual parameter vectors to estimate, subject to subspace constraints that enforce the objectives across the network to lie in a low-dimensional subspace. This constrained formulation includes consensus optimization as a special case, and allows for more general task relatedness models such as smoothness. While such formulations can be solved via projected gradient descent, the resulting algorithm is not distributed. Motivated by the centralized solution, we propose an iterative and distributed implementation of the projection step, which runs in parallel with the gradient descent update. We establish that, for small step-sizes mu, the proposed distributed adaptive strategy leads to small estimation errors on the order of mu
Adaptation and Learning Over Networks Under Subspace Constraints-Part I: Stability Analysis
This paper considers optimization problems over networks where agents have individual objectives to meet, or individual parameter vectors to estimate, subject to subspace constraints that require the objectives across the network to lie in low-dimensional subspaces. This constrained formulation includes consensus optimization as a special case, and allows for more general task relatedness models such as smoothness. While such formulations can be solved via projected gradient descent, the resulting algorithm is not distributed. Starting from the centralized solution, we propose an iterative and distributed implementation of the projection step, which runs in parallel with the stochastic gradient descent update. We establish in this Part I of the work that, for small step-sizes , the proposed distributed adaptive strategy leads to small estimation errors on the order of . We examine in the accompanying Part II (R. Nassif, S. Vlaski, and A. H. Sayed, 2019) the steady-state performance. The results will reveal explicitly the influence of the gradient noise, data characteristics, and subspace constraints, on the network performance. The results will also show that in the small step-size regime, the iterates generated by the distributed algorithm achieve the centralized steady-state performance